feat: add global system tables framework under `sys` database by suxiaogang223 · Pull Request #366 · alibaba/paimon-cpp

suxiaogang223 · 2026-06-13T16:45:47Z

Summary

Add global system table support under the sys database, enabling catalog-level metadata queries.

Closes part of #141.

Changes

Infrastructure

Add GetOptions() to Catalog interface for catalog-level config access
FileSystemCatalog stores and exposes catalog options
New GlobalSystemTableLoader with independent registry for sys tables
Extend SystemTablePath with is_global flag and TryParsePath to detect sys/ paths
FileSystemCatalog handles sys database in ListTables/DatabaseExists/TableExists/LoadTableSchema

Global System Tables

Table	Status	Columns
`sys.catalog_options`	✅ Complete	`key`, `value`
`sys.all_table_options`	✅ Complete	`database_name`, `table_name`, `key`, `value`
`sys.tables`	✅ Complete	`database_name`, `table_name`, `table_type`, `partitioned`, `primary_key`, `record_count`, `file_size_in_bytes`, `file_count`, `last_file_creation_time`
`sys.partitions`	✅ Complete	`database_name`, `table_name`, `partition_name`, `record_count`, `file_size_in_bytes`, `file_count`, `last_update_time`

Integration Tests (4 new tests in SystemTableReadInteTest)

TestReadGlobalCatalogOptions — verifies schema + content
TestReadGlobalAllTableOptions — verifies table options across databases
TestReadGlobalTables — verifies schema, table metadata
TestReadGlobalPartitions — verifies empty result for unpartitioned tables

Files Changed

 include/paimon/catalog/catalog.h                   |   5 +
 src/paimon/CMakeLists.txt                          |   1 +
 src/paimon/core/catalog/catalog.cpp                |   2 +-
 src/paimon/core/catalog/file_system_catalog.cpp    |  39 +-
 src/paimon/core/catalog/file_system_catalog.h      |   6 +-
 .../core/table/system/global_system_tables.cpp     | 370 ++++++++++++++
 .../core/table/system/global_system_tables.h       | 117 +++++
 src/paimon/core/table/system/system_table.cpp      |  28 +-
 src/paimon/core/table/system/system_table.h        |   2 +
 .../core/catalog/file_system_catalog_test.cpp      |  14 +-
 test/inte/read_inte_test.cpp                       | 237 +++++++++
 11 files changed, 810 insertions(+), 10 deletions(-)

Verification

Fedora x86_64, GCC 16:

Build: ✅
paimon-core-test: 36/40 pass (4 failures pre-existing)
paimon-read-inte-test (global tables): 4/4 ✅
paimon-read-inte-test (system tables): same as before

🤖 Generated with Claude Code

zjw1111

Thanks a lot for building out the global sys system tables — the loader/registry split and reuse of InMemorySystemTable make the design clean and easy to follow.

I left a few inline comments. The most important one is a potential crash on the normal engine read path for sys.tables / sys.partitions / sys.all_table_options (see the comment in system_table.cpp). A couple of smaller notes are below:

global_system_tables.cpp (CatalogOptionsSystemTable::BuildRows, around line 264): this table stores fields via a non-owning std::string_view into context_.catalog_options, while the other tables copy through BinaryString (StringValue). It's safe today because the map is a stable member, but for consistency and to avoid a future dangling-view footgun, would it be possible to use StringValue(...) here too?
file_system_catalog.cpp (line 282): the const_cast<FileSystemCatalog*>(this) looks unnecessary since only const Catalog methods are used later — declaring GlobalSystemTableContext::catalog as const Catalog* would let you drop it. Minor, feel free to skip.

Thanks again!

zjw1111

One additional inline note on the sys.tables primary_key column — a schema/data-contract nit. Thanks!

- Add GetOptions() to Catalog interface for catalog-level config access - FileSystemCatalog stores and exposes catalog_options - New GlobalSystemTableLoader with independent registry for sys tables - Implement sys.catalog_options, sys.all_table_options, sys.tables - Stub for sys.partitions (manifest aggregation to follow) - Extend SystemTablePath with is_global flag - TryParsePath detects sys/ paths for TableScan/TableRead routing - FileSystemCatalog handles sys database in ListTables, DatabaseExists, TableExists, LoadTableSchema Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ructor

…dent fields

Replace the TODO stubs in TablesSystemTable::BuildRows() with actual manifest entry aggregation. The new AggregateFileStats() helper reads the latest snapshot data files and computes record_count, file_size, file_count, and last_file_creation_time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Replace the stub with actual partition-level file statistics using the AggregateFileStats helper. For each partitioned table, read manifest entries and emit one row per partition with record_count, file_size, file_count, and last_update_time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add 4 tests to SystemTableReadInteTest: - TestReadGlobalCatalogOptions: verifies sys.catalog_options schema and content - TestReadGlobalAllTableOptions: verifies sys.all_table_options with table options - TestReadGlobalTables: verifies sys.tables schema, table_type, partitioned, pk - TestReadGlobalPartitions: verifies sys.partitions returns empty for unpartitioned Tests use a ReadGlobalSystemTable helper that creates the GlobalSystemTableContext with a proper Catalog pointer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

suxiaogang223 mentioned this pull request Jun 13, 2026

fix: isolate external projects from host toolchain library paths #367

Open

suxiaogang223 force-pushed the codex/global-system-tables branch from e4597fa to 92de892 Compare July 2, 2026 08:45

suxiaogang223 marked this pull request as ready for review July 2, 2026 12:29

suxiaogang223 force-pushed the codex/global-system-tables branch from 55828f8 to c5d3439 Compare July 2, 2026 13:00

zjw1111 reviewed Jul 9, 2026

View reviewed changes

Comment thread src/paimon/core/table/system/system_table.cpp

Comment thread src/paimon/core/table/system/global_system_tables.h Outdated

Comment thread src/paimon/core/table/system/system_table.cpp Outdated

zjw1111 reviewed Jul 9, 2026

View reviewed changes

Comment thread src/paimon/core/table/system/global_system_tables.cpp Outdated

lxy-9602 reviewed Jul 9, 2026

View reviewed changes

suxiaogang223 force-pushed the codex/global-system-tables branch from 76f878a to d79ca7b Compare July 13, 2026 11:41

suxiaogang223 and others added 13 commits July 17, 2026 10:27

fix: remove unused helper functions in global system tables

dd0dab9

fix: replace ValueUnsafe() with value() in global system tables

bac294d

fix: add default value for catalog_options in FileSystemCatalog const…

143c550

…ructor

fix: update TestInvalidList to expect global system table names

ac0bce0

improve: add table_type detection and clearer TODO for manifest-depen…

c1c03a9

…dent fields

style: apply clang-format fixes from pre-commit

31f52b5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix: address global system table review feedback

d7192b2

fix: complete global system table review feedback

0402ff6

test: configure bucket key for append-only table

fc812bb

suxiaogang223 force-pushed the codex/global-system-tables branch from bc70c46 to fc812bb Compare July 17, 2026 02:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add global system tables framework under `sys` database#366

feat: add global system tables framework under `sys` database#366
suxiaogang223 wants to merge 13 commits into
alibaba:mainfrom
suxiaogang223:codex/global-system-tables

suxiaogang223 commented Jun 13, 2026 •

edited

Loading

Uh oh!

zjw1111 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zjw1111 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

suxiaogang223 commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Infrastructure

Global System Tables

Integration Tests (4 new tests in SystemTableReadInteTest)

Files Changed

Verification

Uh oh!

zjw1111 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zjw1111 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

suxiaogang223 commented Jun 13, 2026 •

edited

Loading